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The  Air  Traffic  Control  Operational  Error  Severity  Index; 

An  Initial  Evaluation 


Currently,  the  FAA  Air  Traffic  Investigations  Division 
oversees  and  coordinates  the  Operational  Error  (OE)  re¬ 
porting  process  governed  by  the  FAA  Air  Traffic  Quality 
Assurance  Order  7210.56c  (FAA,  2002)  that  details  the 
conduct  of  incident  reporting  processes.  The  need  for  a 
formal,  system-error  reporting  program  was  recognized 
early  in  the  FAAs  evolution.  Recommendations  were 
developed  regarding  the  conduct  of  incident  investiga¬ 
tions  and  the  use  of  the  resulting  information,  many 
of  which  were  incorporated  into  the  FAA  Order  8020 
(O’Connor  &  Pearson,  1965).  In  1964,  a  six-month  test 
of  an  expanded  operational  error/operational  deviation 
(OE/OD)  reporting  form  was  conducted  at  two  en  route 
centers  located  in  Miami  andjacksonville,  Florida.  Byjuly 
1965,  FAA  Order  8020,  the  AT  System  Error  Reporting 
Program,  was  revised  “to  provide  more  complete  error  oc¬ 
currence  information  and  simplify  typing  with  a  resultant 
savings  in  time”  (League,  1965).  In  addition  to  many 
items  resembling  those  used  in  the  current  report,  this 
form  included  a  matrix  of  points  assigned  to  horizontal 
and  vertical  separation.  This  information  was  used  to 
categorize  the  event  as  major,  moderate,  or  minor,  using 
horizontal  and  vertical  proximity.  Although  this  matrix 
was  later  discontinued,  the  system  was  continually  revised 
to  become  the  current  operational  error  reporting  system 
(FAA  Order  7210.56c).  The  error  reporting  system  has 
demonstrated  the  relatively  infrequent  nature  of  OEs. 
For  example,  in  fiscal  year  2000  alone,  the  U.S.  air  traf¬ 
fic  system  handled  166,669,557  operations  with  an  OE 
rate  of  .69  per  1000,000  facility  activities.  (FAA,  2002). 
However,  concern  has  been  expressed  that  the  error  rate 
has  gradually  increased  from  .49  in  1997. 

In  response  to  a  recommendation  in  the  2000  report 
by  the  U.S.  Department  of  Transportation’s  Office  of 
Inspector  General  (OIG),  the  FAA  developed  a  method 
to  assign  severity  values  for  operational  errors  that  occur  in 
flight.  A  memorandum  of  understanding  was  negotiated 
between  the  National  Air  Traffic  GontroIIers  Association 
(NATGA)  and  the  FAA  to  support  this  effort.  Having 
a  tool  to  group  OEs  by  severity  would  facilitate  the  pri¬ 
oritization  and  allocation  of  resources  to  focus  attention 
more  efficiently  on  OEs  in  the  different  severity  groups. 
Later,  in  2003,  an  OIG  report  (hereafter  called  the  OIG 
report)  led  to  a  scientific  evaluation  of  the  OE  Severity 
Index  (SI)  and  its  component  parts.  We  report  on  the 
results  of  that  evaluation. 


Operational  Error  Severity  Index  (SI) 

A  study  of  OEs  throughout  the  National  Airspace 
System  (NAS)  served  as  the  basis  for  development  of  a 
model  that  would  characterize  their  severity  (Rodgers  & 
Nye,  1993).  Information  from  the  initial  investigation 
was  used  as  the  basis  for  development  of  the  current 
SI.  The  index  is  based  on  the  following  factors:  vertical 
and  horizontal  separation  distances,  relative  flight  paths, 
cumulative  closure  rates,  and  the  level  of  ATG  awareness 
(hereafter  referred  to  as  ATG  Gontrol).  Since  these  factors 
are  based  on  operationally  relevant  criteria  associated  with 
the  provision  of  ATG  services,  they  allow  for  the  calcula¬ 
tion  of  the  severity  of  air  traffic  control  separation  losses. 
These  factors  are  allocated  various  point  values  that  total 
100  and  thereby  allow  for  an  assessment  of  relative  safety 
across  operational  errors. 

The  current  Severity  Index  formula  is  computed  from 
data  that,  for  the  most  part,  can  be  objectively  determined 
by  post  hoc  incident  investigation.  Points  are  assigned  for 
varying  levels  of  vertical  separation,  horizontal  separation, 
closure  rate,  flight  paths,  and  the  amount  of  control. 
Determination  of  whether  the  event  was  controlled  or 
uncontrolled  is  the  only  variable  that  requires  ATG  expert 
judgment  to  determine  a  point  value. 

Vertical  and  Horizontal  separation  are  defined  as  the 
minimum  separation  based  on  the  radar  data  just  prior 
to  aircraft  divergence.  Together,  these  factors  make  up 
to  50%  of  the  SI  point  total.  As  shown  in  Tables  la 
and  lb,  the  maximum  point  values  (25  points  each)  are 
assigned  for  both  vertical  and  horizontal  distances  for 
events  that  culminate  in  proximities  of  less  than  500 
ft  vertical  and  less  than  D-mile  horizontal  separation. 
Given  existing  radar  tolerances  and  measurement  tools, 
further  division  of  minimum  separation  distance  would 
be  of  limited  value.  Accepting  the  notion  that  the  safety 
margin  decreases  geometrically,  as  opposed  to  a  steady 
arithmetic  decline  as  aircraft  get  closer,  points  assessed 
for  close  encounters  remain  higher.  Gonversely,  as  vertical 
and  horizontal  separation  increase,  the  potential  threat  to 
safety  is  assumed  to  be  lower,  and  their  associated  point 
assignments  drop  off  rapidly. 

When  wake  turbulence  separation  standards  govern 
aircraft  around  an  airport,  SI  point  values  for  vertical 
separation  are  not  calculated.  Instead,  in-trail  separa¬ 
tions  point  values  are  used,  as  shown  at  the  bottom  of 
Table  lb. 
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Respective  point  values  for  closure  rates  and  flight  paths 
were  developed  to  objectively  include  these  parameters 
as  components  of  the  Severity  Index  (Tables  la- lb).  It 
should  be  noted  that  faster  aircraft  closure  rates,  coupled 
with  converging,  opposite-direction  flight  paths  reduce 
the  safety  margin  more  so  than  slower  closure  rates  and 
diverging  flight  paths.  Also,  a  significant  assumption 
is  made  that  aircraft  on  diverging  or  non-intersecting 
paths  have  little  impact  on  the  overall  safety  margin  and 
consequently  do  not  receive  Severity  Index  points.  How¬ 
ever,  head-on  encounters  at  high  rates  of  speed,  coupled 
with  minimum  radar  data  separation  distance  prior  to 
divergence,  account  for  the  greatest  total  point  value. 
The  highest  possible  point  values  assigned  to  closure  rate 
and  flight  path  determinations  make-up  10%  and  20%, 
respectively,  of  the  index. 

A  final  factor,  ATC  Control,  completes  the  assessment 
of  severity  for  each  airborne  operational  error.  While  the 
main  purpose  of  the  ATC  system  is  to  preserve  safety, 
determining  the  adequacy  of  ATC  services  assists  in  un¬ 
derstanding  shortfalls  that  occasionally  occur;  therefore, 
ATC  Control  is  included  in  the  model  of  error  severity.  As 
with  the  other  factors,  an  initial  assumption  is  made  that 
maximum  point  values  are  assigned  when  the  controller 
was  unaware  of  the  conflict,  took  no  corrective  action, 
and/or  became  aware  of  the  conflict  but  did  not  have 
sufficient  time  to  effectively  mitigate  the  loss  of  separa¬ 
tion  (20%  of  maximum  index  score).  For  encounters  in 
which  a  controller  is  aware  of  the  impending  conflict  and 
issues  control  instructions  insufficient  to  attain  required 
aircraft  separation  distances,  minimum  point  values  are 
assigned  for  this  factor,  as  it  is  considered  a  controlled 
event  (4%  of  maximum  index  score).  For  events  in  which 
a  Traffic  Alert  and  Collision  Avoidance  System  Resolu¬ 
tion  Advisory  (TCAS-  RA)  is  initiated  and  the  controller 
was  taking  actions  to  mitigate  the  event,  an  intermediate 
point  value  is  assigned  (15%  of  maximum  index  score). 
The  ATC  Control  component  of  the  Severity  Index  was 
introduced  in  April  of  2002. 

While  an  initial  estimate  of  the  overall  point  total  for 
an  OE  is  determined  as  part  of  the  preliminary  report,  a 
specialist  at  FAA  Headquarters  reviews  each  of  the  events 
and  assigns  the  final  rating.  The  point  total  for  an  event 
can  range  from  12  through  100.  As  a  part  of  this  effort, 
three  severity  levels  were  initially  established.  High  Severity 
events  have  a  point  range  from  90  to  1 00.  Moderate  events 
range  from  40  to  89  points,  and  Low  Severity  events  have 
a  rating  of  39  or  less.  Later,  the  Moderate  category  was 
further  divided  into  Low  and  High  Moderate,  primarily 
based  on  whether  the  controller  was  aware  (Low  Moderate) 
or  not  aware  (High  Moderate)  that  an  error  was  occurring 
at  the  time  of  the  closest  loss  of  separation. 


The  following  is  an  example  of  how  the  point  total 
is  used.  If  an  aircraft  is  in  en  route  airspace  at  a  higher 
altitude,  standard  separation  calls  for  the  minimum  sepa¬ 
ration  to  involve  either  a  horizontal  distance  of  5  miles 
or  a  vertical  distance  of  2,000  ft.  As  separation  between 
aircraft  is  reduced  to  less  than  the  required  distance,  points 
are  assigned  for  the  moment  when  minimum  separation 
occurred.  If  the  aircraft  pair  was  separated  by  less  than 
500  ft  vertically  (25  points)  and  between  1  to  1.499  miles 
(24  points),  the  subtotal  would  reach  49  points.  At  a 
closure  rate  of  300  to  699  knots,  an  additional  8  points 
would  be  added.  If  the  flight  paths  were  converging  on  a 
crossing  course,  another  1 8  points  would  be  added.  The 
final  factor  included  in  the  algorithm  involves  a  decision 
regarding  the  controller’s  awareness  of  the  event.  If  it  was 
uncontrolled,  a  situation  where  the  controller  was  not 
taking  action  to  avoid  the  loss  of  separation,  another  20 
points  would  be  added.  This  is  the  only  factor  of  the 
Severity  Index  where  subjective  judgment  plays  a  role. 
All  other  information  is  available  from  objective  sources. 
For  this  particular  event,  the  point  total  is  95,  placing 
this  event  in  the  High  Severity  category 

As  part  of  an  evaluation  of  the  FAAs  SI,  this  report 
focuses  on  three  objectives:  (1)  to  evaluate  the  SI  and 
identify  the  relationships  between  components  of  the  SI, 
(2)  to  examine  the  aircraft  safety  margin  associated  with 
SI  point  values,  and  (3)  to  examine  the  cut  score  used  to 
differentiate  between  High  Moderate  and  High  Severity 
OEs.  The  analyses  were  designed  to  evaluate  the  rela¬ 
tionships  between  loss  of  separation  from  the  prescribed 
standards  and  the  Severity  Index,  and  to  identify  ways  in 
which  the  index  can  be  used  to  support  efforts  to  develop 
strategies  that  will  mitigate  the  severity  of  OEs. 

METHODS  AND  PROCEDURES 

Index  Scores 

The  OE  database  contained  2,390  OEs  that  occurred 
between  April  200 1  and  May 2003.  Missing  values  in  394 
cases  left  a  total  of  1,996  OEs  with  valid  index  scores. 
Of  the  1,996  cases,  572  involved  the  En  Route  2,000  ft 
requirement,  723  for  the  En  Route  1 ,000  ft  requirement, 
679  were  Terminal  and  En  Route  Single  Site  Radar  (TE- 
Single  Site),  and  22  were  Wake  Turbulence. 

Severity  Index  Variables 

The  OE  SI  is  a  function  of  five  variables.  Vertical 
Separation  (feet).  Horizontal  Separation  (miles).  Closure 
Rate  (knots).  Flight  Path  (converging  -  opposite  course, 
converging  -  crossing  course,  same  course,  diverging/ 
nonintersecting),  and  ATC  Control  (Uncontrolled,  Con¬ 
trolled  no  TCAS  RA,  and  Controlled  with  TCAS  RA). 
The  only  subjective  component  is  ATC  Control. 
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Statistical  Analysis 

Descriptive  statistics  were  used  to  assess  the  distribution 
of  the  SI  components.  Correlations  between  each  of  the 
rating  categories  that  comprise  the  SI  and  the  SI  scores  were 
computed  to  better  understand  the  relationships  between 
the  ratings  and  the  overall  SI  score.  For  these  analyses, 
we  used  the  actual  vertical  and  horizontal  separation  dis¬ 
tances  and  the  ratings  for  the  closure  rate  category.  Flight 
Paths  were  categorized  as  follows:  Converging-  Opposite 
Courses  (1),  Converging  -  Crossing  Courses  (2),  Same 
Course  (3),  and  Diverging/Non-Intersecting  (4).  The 
ATC  Control  Factor  events  were  categorized  in  a  similar 
fashion;  Uncontrolled  (1),  Controlled  with  TCAS  RA 
(2),  and  Controlled  with  no  TCAS  RA  (3). 

To  examine  the  aircraft  safety  margin,  ten  loss-of- 
separation  categories  were  created  based  on  the  percent¬ 
age  of  available  separation  (10%  or  less  through  91%  to 
100%  of  the  separation  standard)  for  horizontal  and  for 
vertical  separation.  Next,  we  determined  the  number  and 
percentage  of  the  OEs  that  fell  within  each  of  the  ten 
loss-of-separation  categories.  These  were  determined  for 
each  of  the  three  separation  standards  (En  Route  1,000 
ft,  En  Route  2,000  ft,  and  Terminal  and  En  Route  Single 
Site  Radar).  Finally,  we  calculated  the  average  SI  score  for 
the  OEs  that  were  present  in  each  of  the  ten  categories 
defined  above.  To  illustrate  these  differences  graphically, 
we  collapsed  the  ten  loss-of- vertical  separation  categories 
into  five  categories  to  simplify  the  illustration.  We  then 
plotted  the  percentage  of  OEs  in  those  categories  against 
the  ten  horizontal  loss-of-separation  categories. 

Discriminant  analyses  were  conducted  to  determine 
whether  there  was  subjective  bias  associated  with  dif¬ 
ferentiating  between  Fligh  Moderate  and  Fligh  Severity 
OEs.  The  goal  of  the  discriminant  analysis  was  to  develop 
an  equation  that  classified  OEs  into  these  two  categories 
based  on  the  SI  components.  The  classification  results 
were  then  compared  with  the  results  of  classifying  OEs 
using  the  SI.  A  high  degree  of  correspondence  between 
the  comparisons  would  indicate  that  the  SI  is  consistent 
with  the  objective  data.  Similarly,  a  low  correspondence 
between  the  comparisons  would  indicated  that  the  SI 
was  inconsistent  with  the  objective  data.  Moreover,  the 
weights  assigned  to  each  of  the  predictors  were  compared 
with  the  weights  used  in  the  current  SI.  A  high  degree  of 
correspondence  between  the  comparisons  would  indicate 
that  the  SI  system  of  weighting  was  consistent  with  em¬ 
pirically  derived  weights.  Objective  values  of  vertical  and 
horizontal  separation,  closure  rate,  and  flight  path,  along 
with  a  subjective  measure  of  ATC  Control  (no  objective 
measure  exists),  were  used  to  classify  OEs.  Flowever,  in 
contrast  to  the  values  used  in  computing  correlations,  the 
ATC  Control  factor  was  dichotomized:  1  =  controlled. 


and  2  =  uncontrolled.  Separate  analyses  were  conducted 
for  the  three  separation  conditions:  En  Route  2,000  ft, 
En  Route  1 ,000  ft,  and  TE-  Single  Site. 

It  should  be  noted  that  the  results  of  the  discriminant 
analyses  cannot  be  directly  compared  with  the  results  of 
the  correlations.  This  is  because  the  two  statistics  are  com¬ 
puted  over  different  portions  of  the  SI.  The  correlations 
are  calculated  across  the  entire  SI  range.  In  contrast,  the 
discriminant  analyses  are  conducted  for  SI  values  greater 
that  79.  Since  the  area  of  interest  for  the  discriminant 
analysis  is  the  cut  score  separating  High  Moderate  OEs 
from  High  Severity  OEs,  it  is  important  that  the  focus  of 
the  discriminate  function  should  be  on  OEs  within  the 
region  of  dispute.  If  the  total  range  of  High  Moderate 
(70-89)  is  used,  the  classification  results  will  be  inflated 
due  to  the  high  base  rate  associated  with  this  category. 

RESULTS 
Descriptive  Statistics 

Descriptive  information  regarding  the  components  of 
the  OE  SI  is  provided  in  Table  2.  There  was  considerable 
variability  in  vertical  separation,  mean  of  621  ft  and  SD 
of  484.49.  In  contrast,  the  horizontal  separation  had  a 
SD  of  1.34  around  the  mean  of  2.932  miles. 

Nearly  one-half  of  the  OEs  involved  a  Diverging/ 
Non-Intersecting  flight  path  where  no  points  were  added 
for  either  closure  rate  or  flight  path.  With  the  exception  of 
OEs  where  the  closure  rate  exceeded  699  knots  (2.5 1%), 
the  percentage  involving  the  three  other  closure  rate  cat¬ 
egories  was  relatively  consistent,  ranging  from  13.53% 
to  18.69%  of  the  OEs.  Only  a  small  percentage  of  the 
OEs  involved  converging,  opposite-direction  flight  paths 
(5.21%)  or  Same  Course  headings  (9. 12%).  Nearly  75% 
were  considered  to  be  Uncontrolled  events  (70.84%).  A 
small  percentage  was  considered  as  a  Controlled  event 
that  involved  a  TCAS  RA  (2.15%).  Diverging  flight 
paths  involved  a  higher  percentage  of  ControIIed-No 
TCAS  RA  events  (42.8%). 

When  severity  scores  are  used  and  only  those  OEs  that 
have  been  assessed  by  this  metric  to  date  utilized,  the 
distribution  reflects  a  somewhat  negative  skew  as  shown 
in  Figure  1.  A  majority  of  the  errors  (73.1%)  fell  in  the 
Moderate  Severity  group.  The  High  Severity  category  in¬ 
volved  5 .47%  of  the  OEs,  and  the  low  comprised  2 1 .44% 
of  the  OEs.  This  is  similar  to  the  sample  used  for  the  OIG 
report:  8%  high,  72%  moderate,  and  22%  low. 

Correlations  between  the  SI  score  and  the  rating 
categories  that  comprise  the  index  under  the  three  sepa¬ 
ration  standards  and  across  all  conditions  (overall)  are 
presented  in  Tables  3a  through  3d.  The  small  number 
of  instances  where  the  OEs  involved  Wake  Turbulence 
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made  the  use  of  those  correlations  of  limited  utility.  For 
the  three  separation  standards,  correlations  between  the 
rating  categories  were  generally  low  to  moderate.  The 
exception  was  the  consistently  high  correlations  between 
Flight  Path  and  Closure  Rate  (ranging  from  -.85  to  -.97). 
This  is  understandable  when  one  considers  that  the  highest 
closure  rates  occur  when  aircraft  are  flying  toward  each 
other  on  opposite  courses.  The  negative  relationship  is 
simply  an  attribute  of  the  coding  system  used  to  classify 
the  rating  categories  in  the  data  analysis.  A  value  of  1 
was  assigned  to  the  highest  Closure  Rate,  and  a  value  of 
4  was  assigned  to  a  Flight  Path  that  was  on  a  diverging/ 
non-intersecting  course.  Another  factor  that  contributed 
to  the  high  correlation  is  the  restricted  range  of  points 
associated  with  Closure  Rate  and  Flight  Path.  Nearly  half 
of  the  OEs  analyzed  received  no  points  for  Closure  Rate 
and  4  points  for  Flight  Path. 

Statistically,  all  of  the  correlations  between  the  SI  rat¬ 
ing  categories  and  the  SI  scores  were  significant  (p<.01). 
Flight  Path  exhibited  the  highest  correlations  with  the  SI 
scores,  ranging  from  a -.77  for  theTE-Single-Site  separa¬ 
tion  condition  to  -.78  for  En  Route  1 ,000  ft  and  2,000  ft 
separation.  Correlations  between  Closure  Rate  and  the  SI 
scores  were  similar  to  those  for  the  Flight  Path  measure 
(ranging  from  .71  to  .77).  Overall,  the  lowest  correlations 
between  the  SI  rating  categories  and  the  SI  scores  were 
found  for  vertical  separation  (ranging  from  a  -.14  to  a 
-.41).  The  higher  correlations  between  the  SI  and  the 
Flight  Path  and  Closure  Rate  variables  suggest  that  they 
are  relatively  more  important  in  determining  the  overall  SI 
score.  Flowever,  the  overlap  of  these  two  variables  suggests 
that  once  the  contribution  of  one  has  been  considered 
there  would  be  little  additional  information  provided  by 
the  other.  Additionally,  correlations  between  horizontal 
separation  and  the  other  SI  measures  were  consistently 
higher  than  those  between  those  measures  and  vertical 
separation.  This  suggests  that  the  horizontal  separation 
dimension  may  play  a  more  prominent  role  in  predicting 
the  SI  score  of  an  OE  than  does  the  vertical  separation. 

Safety  Margin 

Figures  2-4  graphically  illustrate  the  number  of  OEs 
that  fell  within  each  of  the  combined  loss  of  separation 
standards  categories.  A  scan  of  the  En  Route  1 ,000  ft  and 
2,000  ft  separation  figures  (Figs.  2  and  3)  reveals  that  a 
majority  of  the  errors  lie  on  the  right-hand  side  involv¬ 
ing  60%  or  more  of  the  horizontal  separation.  For  the 
TE-Single-Site  separation  condition  (Figure  4)  the  errors 
exhibit  greater  spread  across  the  two  dimensions.  Only  a 
small  number  of  OEs,  from  0.7%  (4/571)  for  En  Route 


2,000  ft  to  10.5%  (71/671)  for  TE-Single-Site  separa¬ 
tion,  involved  30%,  or  less,  of  prescribed  horizontal  and 
vertical  separation.  As  one  could  expect,  due  perhaps  to 
the  more  restricted  range  of  altitudes  available  for  vertical 
separation,  there  was  a  higher  percentage  of  OEs  under 
the  TE-Single-Site  separation  condition  that  involved 
less  than  50%  of  the  prescribed  standards  (36.3%)  when 
compared  with  the  En  Route  1,000  ft  (12.3%)  and  En 
Route  2,000  ft  conditions  (7.2%). 

Figures  5  and  6  provide  information  regarding  the 
average  SI  score  based  on  percent  of  the  required  hori¬ 
zontal  or  vertical  separation.  With  the  exception  of  the 
values  for  the  Wake  Turbulence  condition,  the  mean  SI 
score  declined  linearly  from  an  average  of  80  to  90  for 
the  three  separation  conditions  to  an  average  of  36  to  55 
for  horizontal  separation  (Fig.  5). 

Changes  in  the  average  SI  score  for  percent  of  vertical 
separation  presented  a  much  different  picture  (Fig.  6).  For 
the  categories  involving  0  to  50%  of  the  required  vertical 
separation,  the  average  SI  scores  for  the  three  separation 
conditions  (En  Route  1,000  ft,  En  Route  2,000  ft,  and 
TE-Single-Site)  remain  separate  and  fairly  stable.  Under 
the  TE-Single-Site  separation  standards,  the  average  SI 
score  ranged  from  76%  to  around  69%  across  the  0- 
10%  through  the  41-50%  categories  of  vertical  separa¬ 
tion.  Across  the  same  categories,  the  average  SI  score 
ranged  from  around  59%  to  63%  for  En  Route  1,000ft 
The  average  SI  for  En  Route  2,000  ft  also  evidenced  a 
slight  increase  from  53  at  1-10%  of  vertical  separation 
to  56  at  51-60%.  From  the  51-60%  percent  of  vertical 
separation,  the  average  SI  score  declined  markedly  to  an 
average  of  41-47  at  90%  and  above.  The  only  exception 
is  a  score  of  68  for  the  TE-Single-Site  condition.  This  is 
the  score  for  a  single  OE  that  fell  within  the  90%  and 
above  category. 

These  comparisons  reveal  the  relatively  greater  impor¬ 
tance  of  loss  of  horizontal  separation  in  determining  the 
overall  SI  score.  SI  scores  increase  in  a  relatively  linear 
fashion  for  each  10%  loss  in  horizontal  separation.  These 
changes  are  similar  for  each  of  the  separation  standards, 
with  the  exception  of  Wake  Turbulence.  Outcomes  sug¬ 
gest  that  once  there  has  been  a  50%  loss  in  the  prescribed 
vertical  separation,  the  continued  loss  ofvertical  separation 
contributes  little  to  any  overall  increase  in  the  SI  score. 
The  results  also  suggest  that  the  loss  of  vertical  separa¬ 
tion  may  have  a  different  influence  in  determining  the 
SI  score  than  horizontal  separation.  Additionally,  loss 
of  vertical  separation  my  operate  somewhat  differently 
under  the  two  En  Route  and  TE-Single-Site  separation 
conditions. 
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Rater  Bias 

Discriminant  Analyses  were  conducted  separately  for 
the  three  separation  standards:  (1)  High  En  Route  (with  a 
2000  ft  requirement,  n  =  73),  (2)  LowEn  Route  (having 
a  1000  ft  requirement,  n  =  126),  and  (3)  Low  En  Route 
(having  a  1000  ft  requirement,  n  =  126),TE-Single-Site 
(n  =  185).  The  predictor  variables  were  Vertical  Separa¬ 
tion  (V),  Horizontal  Separation  (H),  Flight  Path  (FP), 
Closure  Rate  (CR),  and  ATC  Control  (ATCC). 

The  resulting  discriminant  functions  (L)  are  presented 
below. 

High  En  Route:!  =  (1.34)V+(.83)H+(0)FP-  (.78)CR+(0)ATCC. 
LowEn  Route:  L  =  (1.09)V+(1.02)H-(.77)FP-  (.52)CR+(0)ATCC. 
TE-Single-Site:  L  =  (.61)V+(1.03)H-(.14)FP-  (.34)CR+(0)ATCC. 

As  can  be  seen,  the  beta  weights  for  elements  of  the 
SI  that  were  included  in  the  equations  for  the  High  En 
Route  and  Low  En  Route  were  similar.  The  only  differ¬ 
ence  was  that  Flight  Path  played  a  more  important  role 
for  Low  En  Route  than  for  High  En  Route.  Values  for  the 
High  and  Low  En  Route  conditions  differed  substantially 
from  the  values  included  in  the  TE-Single-Site  analysis. 
Horizontal  separation  played  a  much  more  critical  role  for 
the  TE-Single-Site  separation  condition  than  in  either  of 
the  other  conditions.  Overall,  classification  accuracy  was 
85%  for  both  the  Low  and  High  En  Route  conditions. 
For  the  TE-Single-Site  condition,  classification  accuracy 
was  92%.  In  all  instances,  these  values  were  above  those 
expected  from  the  population  base-rate.  It  should  also 
be  noted  that  weights  of  zero  applied  to  ATCC  and  FP 
do  not  mean  that  these  variables  were  not  important 
classification  variables.  It  just  means  that  for  SI  values 
greater  than  79,  these  variables  became  a  constant  and, 
thus,  were  not  useful  for  differentiating  between  High 
Moderate  and  High  Severity  OEs. 

CONCLUSION 

The  analyses  conducted  in  an  initial  evaluation  of  the 
FAAs  Severity  Index  (SI)  reveal  that  a  complex  relation¬ 
ship  exists  among  the  individual  components  compris¬ 
ing  the  SI  and  the  different  separation  standards  within 
which  the  SI  is  applied.  However,  a  consistent  finding 
was  the  importance  of  horizontal  separation  to  the  overall 
SI  score.  The  more  horizontal  separation  there  was,  the 
less  severe  was  the  OE.  This  was  especially  true  when 
using  the  TE-Single-Site  Separation  Standard.  This 
conclusion  was  further  supported  by  the  results  of  the 
discriminant  analyses.  Greater  weight  was  assigned  to 
horizontal  separation  when  classifying  OEs  into  High 
Moderate  and  High  categories  using  the  TE-Single-Site 
Separation  Standard. 


Despite  the  importance  of  horizontal  separation,  the 
results  of  the  discriminant  analyses  revealed  that  improve¬ 
ments  to  the  SI  might  be  obtained  by  developing  a  different 
weighting  system  for  the  three  separation  standards.  For 
example,  in  Low  En  Route,  the  evidence  suggested  that 
vertical  and  horizontal  separation  should  be  weighted  the 
same  (as  reflected  in  the  current  SI).  Within  High  En 
Route,  the  evidence  suggested  that  more  weight  should 
be  given  to  vertical  separation  compared  to  horizontal 
separation.  However,  as  previously  mentioned,  within 
TE-Single-Site  more  weight  should  be  given  to  horizontal 
separation  compared  to  vertical.  It  should  be  noted  that 
this  suggestion  is  based  on  analyses  conducted  for  SI  values 
greater  than  79.  Additional  research  is  needed  to  explore 
this  matter  more  fully  and  to  determine  whether  similar 
results  can  be  achieved  using  the  entire  range  of  the  SI. 

Finally,  this  evaluation  of  the  FAAs  Severity  Index 
revealed  that  the  SI  provides  a  rational  approach  for 
categorizing  the  severity  of  ATC  Operational  Errors. 
Although  questions  remain  as  to  the  SI  cut  scores  used 
to  categorize  OEs,  it  is  recommended  that  they  not  be 
changed  unless  objective  measures  can  be  developed  that 
support  those  changes.  With  the  exception  of  the  ATC 
Control  component,  the  remaining  four  components 
are  objective  and  are  derived  from  performance  charac¬ 
teristics  of  the  aircraft  involved  in  the  OE.  Additional 
research  is  needed  to  determine  if  the  development  of  a 
checklist  for  the  classification  of  the  ATC  Control  factor 
would  ensure  a  higher  consistency  of  classification  across 
individual  controllers. 
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TABLES  AND  FIGURES 


Table  1a.  Radar  OE  severity  index  en  route  chart 


VERTICAL  SEPARATION 
1,000  feet  required 

POINTS 

HORIZONTAL  SEPARATION 
5-mile  separatiou  requiremeut 

POINTS 

500  feet  to  599  feet 

20 

'/2  mile  to  0.999  mile 

25 

600  feet  to  699  feet 

16 

1  mile  to  1.499  miles 

24 

700  feet  to  799  feet 

12 

1.5  miles  to  2  miles 

24 

800  feet  to  899  feet 

6 

2  miles  to  2.49  miles 

23 

900  feet  to  999  feet 

2 

2.5  miles  to  2.999  miles 

22 

VERTICAL  SEPARATION 
2,000  feet  required 

r 

3  miles  to  3.499  miles 

20 

Less  than  500  feet 

25 

3.5  miles  to  3.999  miles 

16 

500  feet  to  599  feet 

25 

4  miles  to  4.499  miles 

10 

600  feet  to  699  feet 

24 

4.5  miles  to  4.999  miles 

5 

700  feet  to  799  feet 

24 

1  CLOSURE  RATE 

POINTS 

800  feet  to  899  feet 

23 

700  knots  and  greater 

10 

900  feet  to  999  feet 

22 

300  knots  to  699  knots 

8 

1,000  feet  to  1,099  feet 

20 

100  knots  to  299  knots 

6 

1,100  feet  to  1,199  feet 

18 

Less  than  100  knots 

4 

1,200  feet  to  1,299  feet 

16 

FLIGHT  PATHS 

POINTS 

1,300  feet  to  1,399  feet 

14 

Converging  -  Opposite  Courses 

20 

1,400  feet  to  1,499  feet 

12 

Converging  -  Crossing  Courses 

18 

1,500  feet  to  1,599  feet 

10 

Same  Course 

10 

1,600  feet  to  1,699  feet 

8 

Diverging/Non-Intersecting 

0 

1,700  feet  to  1,799  feet 

6 

ATC  CONTROL  FACTOR 

POINTS 

1,800  feet  to  1,899  feet 

4 

Uncontrolled 

20 

1,900  feet  10  1,999  feet 

2 

Controlled  with  TCAS  RA 

15 

Controlled  with  no  TCAS  RA 

4 

Standard  separation  in  the  en  route  environment  is  5  nautical  miles  horizontally  and  1 ,000  ft  vertically  up 
29,000  ft,  and  2,000  feet  vertically  above  29,000  feet.  All  references  to  miles  in  this  report  are  nautical 
miles.  A  nautical  mile  is  equivalent  to  6,076  ft. 
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Table  1b.  Radar  OE  severity  index  terminal  and  en  route  single-site  chart  (TE-Single-Site) 


VERTICAL  SEPARATION 

POINTS 

HORIZONTAL  SEPARATION 
3-mile  separation  requirement 

POINTS 

Less  than  500  feet 

25 

Less  than  'A  mile 

25 

500  feet  to  599  feet 

20 

'A  mile  to  0.999  mile 

18 

600  feet  to  699  feet 

16 

1  mile  to  1.499  miles 

14 

700  feet  to  799  feet 

12 

1.5  miles  to  2  miles 

10 

800  feet  to  899  feet 

6 

2  miles  to  2.49  miles 

6 

900  feet  to  999  feet 

2 

2.5  miles  to  2.999  miles 

2 

CLOSURE  RATE 

POINTS 

HORIZONTAL  SEPARATION 
2.5-mile  requirement 

POINTS 

700  knots  and  greater 

10 

Less  than  'A  mile 

25 

300  knots  to  699  knots 

8 

'/2  mile  to  0.999  mile 

20 

100  knots  to  299  knots 

6 

1  mile  to  1.499  miles 

16 

Less  than  100  knots 

4 

1.5  miles  to  1.999  miles 

10 

FLIGHT  PATHS 

POINTS 

2  miles  to  2.499  miles 

4 

Converging  -  Opposite  Courses 

20 

ATC  CONTROL  FACTOR 

POINTS 

Converging  -  Crossing  Courses 

18 

Uneontrolled 

20 

Same  Course 

10 

Controlled  with  TCAS  RA 

15 

Diverging/Non-Interseeting 

0 

Controlled  with  no  TCAS  RA 

4 

*  When  wake  turbulence  separation  standards  are  governing,  Do  Not  include  any  vertical  point  value. 
Instead  use  the  appropriate  in  trail  separation  index  below,  as  well  as  other  applicable  factors”  (FAA  Order 
7210.56,  section  6-1-5). 


IN  TRAIL  SEPARATION 

4-mile  separation  requirement 

POINTS 

IN  TRAIL  SEPARATION 

5-mile  separation  requirement 

POINTS 

3.49  miles  or  less 

60 

4.499  miles  or  less 

60 

3.5  miles  to  3.999  miles 

35 

4.5  miles  to  4.999  miles 

35 

IN  TRAIL  SEPARATION 

4-mile  separation  requirement 

POINTS 

5.499  miles  and  less 

60 

5.5  miles  to  5.999  miles 

35 

Table  2.  Distribution  of  OEs  across  SI  categories 


Variable 

Sub  Category 

Frequency 

Percent 

Mean 

Std.  Deviation 

Vertical  Sep. 

621.14  ft. 

484.49 

Horizontal  Sep. 
Closure  Rate 

700  or  more 

50 

2.51% 

2.93  miles 

1.34 

Knots 

300-699  kts 

270 

13.53% 

100-299  kts 

352 

17.63% 

Less  than  1 00  kts 

373 

18.69% 

(Diverging) 

951 

47.64% 

Flight  Path 

Conv.  -  Opposite 

104 

5.21% 

Conv.  -  Crossing 

777 

38.93% 

Same  Course 

182 

9.12% 

Diverging/Non- 

Intersecting 

933 

46.74% 

ATC  Control 

Uncontrolled 

1,414 

70.84% 

Controlled  - 

43 

2.15% 

TCAS  RA 
Controlled  - 
No  TCAS  RA 

539 

27.00% 

Table  3a.  Correlations  between  SI  measures  for  the  En  Route  1,000  ft  Separation  Standard 


SIVERT 

SIHORZ 

CTEFAC 

FETPATH  CRATE 

SIVERT 

SIIHORZ 

-.118* 

CTLFAC 

-.134* 

.368* 

FLTPATH 

-.047 

.333* 

.300* 

CRATE 

.058 

-.326* 

-.281* 

-.935* 

SI  INDEX 

-.216* 

-.586* 

-.614* 

-.782*  .758* 

*  p<.01 


Table  3b.  Correlations  between  SI  measures  for  the  En  Route  2,000  ft  Separation  Standard 


SIVERT 

SIHORZ 

CTEFAC 

FETPATH  CRATE 

SIVERT 

SIIHORZ 

-.303* 

CTEFAC 

-.048* 

.326* 

FETPATH 

-.145* 

.448* 

.232* 

CRATE 

.152* 

-.439* 

-.222* 

-.971* 

SI  INDEX 

-.144* 

-.589* 

-.623* 

-.782*  .773* 

*  p<.01 
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TableSc.  Correlations  between  SI  measures  for  the  TE-Single-Site  Separation  Standard 


SIVERT 

SIHORZ 

CTEFAC 

FETPATH 

CRATE 

SIVERT 

SIIHORZ 

.266* 

CTLFAC 

.125* 

.350* 

FLTPATH 

.134* 

.476* 

.304* 

CRATE 

-.056 

-.382* 

-.272* 

-.860* 

SI  INDEX 

-.411* 

-.627* 

-.597* 

-.770* 

.713* 

*  p<.01 

Table  3d.  Correlations  between  SI  measures  across  all  conditions 

SIVERT 

SIHORZ 

CTEFAC 

FETPATH 

CRATE 

SIVERT 

SIIHORZ 

.147* 

CTEFAC 

.082* 

.401* 

FETPATH 

.119* 

.539* 

.337* 

CRATE 

-.045 

-.472* 

-.307* 

-.928* 

SI  INDEX 

-.320* 

-.640* 

-.635* 

-.794* 

.766* 

*  p<.01 


Severity  Index  Value 

Figure  1.  Distribution  of  Severity  Index  values. 
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Low  Altitude  En  Route 

I 10-20%^ 

-^30-40% 
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%  Required  Vertical 
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%  of  Required  Horizontal  Separation 


Figure  2.  Distribution  of  operational  errors  (  n=  719)  by  percent  of  required  horizontal  and 
vertical  separation  for  low  altitude  en  route  with  a  1000  ft  vertical  separation  standard. 


High  Altitude  En  Route 


-10-20% 

-  30-40% 

50-60% 
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100 


Figure  3.  Distribution  of  operational  errors  (n  =  571 )  by  percent  of  required  horizontal  and 
vertical  separation  for  high  altitude  en  route  with  a  2000  ft  vertical  separation  standard. 
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Terminal  and  Single  Site  Radar 
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Figure  4.  Distribution  of  operational  errors  (n  =  671 )  by  percent  of  required  horizontal  and 
vertical  separation  for  Terminal  and  Single-Site  Radar  separation  standard. 


En  Route  1000ft 


Figure.  5.  Average  SI  score  based  on  percent  of  prescribed  horizontal  separation. 
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Percent  of  Required  Vertical  Separation 


Figure  6.  Average  SI  score  based  on  percent  of  prescribed  vertical  separation. 
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